This PR

• disables the MLP Fused Ops if the activation function is not SwiGLU. In this implementation, the rules are generated upfront before checking what model is activated.
• fast_quantized_peft is now removed to prevent further confusion
• Update the benchmarks with all the new updates.

Updated Benchmarks

Outliers

Generally we noticed two things

• THe 70B will now OOM with per device batch size 4. This could be due to some changes, and the benches were hovering around 70+G peak util. This is similar to Quantized Peft Benchmark Experiments Run Out of Memory with Non-Zero Lora Dropout #50, and we will track this there
• Noticed that gpt_bigcode got slower with FSDP. The reason for the slowdown is unclear. It seems to be only affecting this case. This is tracked in Slow down observed for BigCode Santa Coder #110

outliers.csv